Problems may arise in the testing of software. Generally, software performance testing can be applied to web based applications and services, encompassing both single server and multi-server (cloud) implementations. Industry standard tools, e.g., HP Performance Center, JMeter, Neoload, LoadRunner, etc., may be used to conduct performance tests. An input traffic profile may be provided by the test domain expert. Conventionally, the domain expert can foretell the number of hits per unit time, at different times of the day and different days of the week, and so on. Capturing these user input data from past data, a performance test engineer may create the input traffic profile.
As the input traffic is injected to the performance test tool, at the output, two or more output metrics may need to be measured simultaneously. Examples of such output metrics include, without limitation: Latency, Throughput, Error Rate, Missed Deadline Percentage, and so on. However, steady state of each one of the outputs may or may not be reached, or the onset of steady state may not be reliably identified. Steady state arrival may be a function of multiple factors, for example, one of which may be the input traffic rate. Complex interdependencies of the system, and interactions with platforms and other external systems (e.g., CPU load, memory availability, network bandwidth, etc) may significantly impact the steady state outcome of the performance test experiment. Thus, currently, performance testers may be unable to reliably determine the overall extent to which a performance test experiment may have reached a steady state, so that the end user may rely on the results of the experiment.
The current industry practice in software tool-based performance testing essentially ignores the notion of steady state arrival—and leaves the steady state determination to the performance test engineer. Existing industrial tools, e.g., HP PC, JMeter, LoadRunner, NeoLoad may not have built-in measurement tools for steady-state determination. Typically, the performance test engineer would determine steady state upon his/her own, manually, by analyzing the output across a time period and making an analytical determination of the steady state.
Currently, performance testing is being carried out by one or more industry standard tools, such as, for example, HP Performance Center™, JMeter™, Neoload™, and LoadRunner™, among others. The performance testing tools are designed to provide an Average value computation of a chosen output metric across a chosen “period of time interval”. The selection of the “period of time interval” is up to the discretion of the performance test engineer, who may not have much knowledge or awareness as how to compute a steady state. In the existing experiments, the performance test engineer (A) selects the entire experiment duration as a single time interval and obtains the Average values of the output metrics to report the same to the users, or (B) provides a time varying plot of the output metrics across time to the users to make the best interpretation thereof. Currently, no guidelines are provided to the performance test engineer as to what the optimal time interval should be in order to evaluate the system for steady state, and then report the output metrics.
Consequently, current performance testing may provide inconsistent results because no steady state may be reached prior to generating test result data. If the underlying system did not reach a steady state—the end generated performance data is at best a transient state data, and the user may not rely on any such data for a steady state performance of the software under test. For example, the performance test engineer may reports inconsistent and/or unreliable test results to the business users.
Another disadvantage of current methods of software testing involves software tests yielding two or more output metrics. Most real-world web application software involves multiple (2 or more) output metrics. For example, web application software tests may report on latency and response time. One challenge of testing and reporting on two or more output metrics (for example, M1, M2, etc.) is when M1 may reach steady state, M2 may not (that is, M2 may be transient), and/or vice versa. The underlying software under test may not converge simultaneously to a steady state arrival for M1 and M2. If the software does reach a steady state for all its Output metrics, then it's a highly desired coincidence—but depending upon the physical systems and platforms and the characteristics of the software—such convergence may never occur.
In such cases, even if the performance test engineer possesses sufficient time and training to measure steady state, the test engineer may be constrained to determine steady of one metric (M1) and not the other metric (M2). The subjectivity involved, if the software under test should be measured and reported when M1 is steady or when M2 is steady, may lead to incorrect and unreliable reporting. The problem may be amplified if there are three or more metrics. Under current methods of measurement, the test engineer would be required to consider all possible combinations of output metrics to determine which one must reach steady state while the rest may not reach steady state at all.